Everyday Objects Turned Into AR Typing Surfaces
Virtual keyboards have long posed challenges for augmented reality users, often slowing down interaction and creating discomfort due to repeated arm elevation. These issues frequently lead to reduced accuracy and physical strain, limiting the practicality of text entry in immersive environments. In response to this ongoing problem, researchers at the University of Texas at Dallas have developed a system that allows individuals to convert ordinary handheld items into functional typing surfaces within AR settings. The technology aims to provide a more natural and efficient alternative to the flat virtual panels typically used for text input.
The new system, known as PropType, overlays a digital keyboard interface onto a physical object while maintaining responsiveness across both flat and curved surfaces. This capability enables users to type on familiar items such as bottles, books, or cups while maintaining immersion in the virtual environment. To demonstrate its flexibility, student researchers produced a video showing the interface working on a variety of everyday props, highlighting its adaptability to different shapes and materials.
PropType has been designed to address limitations associated with external physical keyboards often used with AR devices. These external tools can interrupt immersion and reduce the sense of presence within the virtual world. By using items already in the user’s hand, the new technology preserves a device-free experience while still offering tactile feedback that helps users confirm their keystrokes. This approach reduces dependence on visual checking and supports more intuitive typing.
The research team presented PropType at the ACM CHI Conference on Human Factors in Computing Systems in Yokohama, Japan, where the project earned a Best Paper Honourable Mention. It was later showcased at the ACM Symposium on User Interface Software and Technology in Busan, South Korea, drawing interest from specialists in human-computer interaction. Both events highlighted the potential of PropType to influence future AR interface design, particularly in scenarios requiring mobility or hands-free input.
Developing the system required examining how users naturally grasp different objects. The team observed 16 participants interacting with a variety of props to analyse grip postures and typing movements. This information informed the creation of tailored keyboard layouts for different shapes and sizes. In addition, a custom editing tool was built to allow users to modify keyboard designs and visual settings, making the system more adaptable to personal preferences and interaction styles.
The work forms part of a broader research effort into haptics, focusing on the role of touch and physical feedback in digital interfaces. This area includes studies on vibrations, temperature changes, and other sensory cues that can enhance how individuals perceive virtual environments. The team has previously been recognised for innovations in thermal masking and thermal-tactile integration, receiving multiple awards at major international conferences.
Thermal masking involves altering the brain’s perception of heat by combining thermal and tactile stimuli. In demonstrations, a heat actuator is placed on a user’s arm while a vibration actuator is positioned several centimetres away. When both are triggered simultaneously, the user perceives heat in both locations, despite the actual source being confined to one spot. The group presented these findings at the ACM CHI Conference on Human Factors in Computing Systems in 2024, illustrating how tactile signals can influence thermal perception.
The broader goal of this research is to create interfaces that feel more immersive and lifelike. By studying how different sensory cues interact, the team aims to develop systems that improve realism across virtual reality, medical training, and other simulation-based applications. These efforts continue to push the boundaries of how touch, temperature, and motion can be integrated into digital experiences to deliver environments that appear almost tangible.
